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DYNAMIC ANOMALOUS PIXEL DETECTION AND CORRECTION 

BACKGROUND OF THE INVENTION 
The present invention relates to digital still or video cameras using a solid 
5 state image pickup device and more particularly to methods and systems used to detect 
and correct defective pixels obtained from a solid state image pickup device. 

The foundation of digital imaging is the image sensor. Just as in a 
traditional camera, light enters a digital still or video camera through a lens controlled by 
a shutter. When the shutter opens, rather than exposing film, the digital camera collects 
10 light on an image sensor - a solid state electronic device. Image sensors contain a grid of 
microscopic photosites or pixels. In a camera, each photosite on the image sensor may 
represents one pixel of a digitized image. Two types of image sensors that are quite 
prevalent in the digital imaging field are the charge-coupled device (CCD) and the 
complimentary metal oxide semiconductor (CMOS) image sensor arrays. These image 
15 sensors may contain anywhere from a few hundred to millions of photosites to convert 
light that is shining on them into small electric currents which can be electronically 
processed and interpreted and eventually used to recreate and display a target image on a 
display device. 

The processes used to produce these image sensors though highly 
20 advanced are not perfect. Photosites in an image sensor array can vary in their sensitivity 
or ability to convert light into electric currents. Some variation is expected and tolerable, 
but on occasion, individual photosites in an array can be dead, inactive or over active. 
This can result in a displayed image having a dark or black dot, bright white dot or 
incorrect color dot, all of which are unwanted and are undesirable to the end user. Those 
25 in the digital imaging field have also recognized the problems caused by improper 

digitization of target images and have proposed various solutions to detect and correct 
defective pixels. These prior art methods and systems can be broadly classified into two 
groups. 

On one hand, there are various hardware-based methods for detecting and 
30 correcting defective pixels taken from an image sensor array. These hardware-based 
methods and systems are quite common to video camera manufacturers. Most 
conventional video cameras using a solid state image pickup device, incorporate a 
defective pixel detection and correction circuit for correcting defective pixels taken from 




an image sensor array. The defective pixels are produced when or after the image sensor 
array is manufactured. Li such a camera, a defective pixel correction circuit detects a 
defective pixel and stores the position data and various data related to the defective pixel 
in a read only memory (ROM) or the like. Then, when the video camera is in use, pixel 
5 data from the defective pixel is replaced by data from a pixel near the defective pixel. 
One such method and system is disclosed in U.S. Pat. No. 5,796,430. The disadvantage 
of such approaches is the need to incorporate and program memory devices during 
product assembly and testing which add costs and delays. Additionally, the hardware 
device costs will also increase due to the need to add the correcting circuitry or logic to 
10 the video processing application specific integrated circuit (ASIC). 

On the other hand, there are various software-based methods for detecting 
i3 and correcting defective pixels in digital images. One such method and system is 

p disclosed in U.S. Pat. No. 5,982,946. Such software-based methods are generally aimed 

at correcting bad image pixels in an already digitized and fiilly processed image. These 
15 already digitized images are in their final displayable and stored forms which have 
g; already been through color processing, compression and various other processing that are 

performed on the raw data which is read off an image sensor array. Therefore, such 
IIJ software-based methods which treat completely processed digital images, may correct 

\ anomalous pixels that are caused by any number of artifacts such as dust or dirt on the 

0 20 original scene, dust or dirt on the lens which was used to capture the scene as well as 
anomalous pixels in the digitized image which were caused by defective photosites. 
These methods typically rely on rather sophisticated and expensive systems and computer 
software to detect and correct bad pixels. These methods generally require user input to 
detect the location of the potentially bad image pixels. Once the user has visually 
25 scanned an entire image and flagged potentially bad image pixels, the system 

incorporating the software program takes over to correct the flagged defective pixels. 
Besides requiring user input, such approaches are also expensive, tedious and very time 
consuming. 

The proliferation of inexpensive PC-interfacing digital still and video 
30 camera devices requires rapid, dynamic, inexpensive and intelligent defective pixel 
detection and correction solutions. 
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SUMMARY OF THE INVENTION 

The present invention provides a method and a system for dynamically 
detecting and correcting anomalous pixels in the raw data taken from an image sensor 
array such as a CCD or a CMOS sensor array, thus allowing the use of dumb cameras to 
capture digital images for subsequent use by an intelligent host - such as being displayed 
on a computer monitor. This invention uses software algorithms running on an intelligent 
host processor to dynamically correct the anomalous pixels in the raw data taken from an 
image sensor array typical of those in a digital still or a video camera. The raw image 
sensor data is the unprocessed brightness data output obtained from the image sensor and 
which has not gone through any lossy compression or color processing. The image 
sensor reads analog voltage or current, converts to digital and sends the signal to a host 
with no further processing or compression. The photosites on the image sensor are used 
to capture either color or monochrome digital still or video images. The raw image data 
is sent to the intelligent host over a bus with a data transfer rate which is determined by 
the bus protocol of the particular bus such as a universal serial bus (USB) or a parallel 
port. All image processing is moved from the camera to the intelligent host. Using the 
combination of a dumb camera which provides raw data to an intelligent host, which does 
all the subsequent image processing, the system works by scanning an image frame for 
pixels that vary more than a specified amount in their brightness value from their 
neighboring pixels and designating those as defective pixels. The location and frequency 
of the photosites sending the defective pixels are stored in a statistical database in the 
computer's memory. The brightness value of a defective pixel is then replaced by a local 
brightness value obtained from the defective pixel's neighboring pixels. The process 
includes video subsampling, meaning that the defective pixel detection is carried out and 
repeated at a pre-specified frame rate to ensure optimum detection and correction at a 
minimal level of scanning. A statistical database is kept so that truly anomalous pixels 
can over time be distinguished from false detection of true anomalies in the target image, 
lighting or other environmentally induced anomalies. 

The software algorithm is intentionally kept very simple so that the 
processor load and/or the video frame rate is minimally affected. First, the software 
algorithm is kept simple because it only performs three fiinctions, namely the detection, 
correction and statistics fiinctions. The statistics routine is kept simple because only 
position information and frequency of occurrence of anomalous pixels are tracked. 
Second, the software algorithm is kept simple so as to have a minimal impact on the rate 



at which data is transferred to the host, so that while a frame is being scanned for 
defective pixels, subsequent frames are not held up. The maximum impact of the 
software algorithm is to at worst reduce the video data transfer rate from 1 0 frames per 
second (fps) to 9 ^s. Third, the software algorithm is kept simple such that the host 
processor's load is not increased by more 1 % when executing the algorithm. The 
anomalous pixels are detected in one frame and the corrections are later instituted on 
subsequent frames as they are read off the image sensor. 

Dynamic defective pixel detection and correction is important for a 
number of reasons. First and foremost, image quality can be significantly improved and 
second, sensor cost can be substantially decreased by increasing the yield or usability of 
otherwise scrap sensors. Dynamic intelligent host software application based anomalous 
pixel detection and correction has several additional cost and quality benefits over the 
traditional hardware-based approaches. First, it reduces manufacturing cost by not having 
to add and program memory during product assembly and test. Second, it allows for 
fiarther reductions in cost by not including the correction logic in the video processing 
ASIC. Finally, it allows for the correction of raw pixel data taken from sensor photosites 
which may become defective over time and with operating environment changes. 

BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 is a simplified block diagram of a)system according to one 

embodiment of the present invention. 

FIG. 2 illustrates a simplified flow chart of a method of detecting and 

correcting defective pixels according to one embodiment of the present invention. 

DESCRIPTION OF THE SPECIFIC EMBODIMENTS 
FIG. 1 shows an embodiment of the present invention which uses software 
algorithms running on an intelligent host processor 1 14 to dynamically correct defective 
or anomalous pixels. In one embodiment, the intelligent host is a server or a PC. The 
system includes an image sensor 1 10 to capture an image, an intelligent host 101 having a 
processor 1 14 to execute the computer program stored in storage 118, and a system bus 
112 to facilitate device communications. The image sensor array can either be a charge- 
coupled device (CCD) or a complimentary metal oxide semiconductor (CMOS) image 
sensor array, which are typically parts of a video or still digital camera. The image sensor 
110 cormnunicates with the host 101 via an external bus 111, which can be a universal 



serial bus (USB) or a parallel port. Upon transfer of appropriate calls between the 
processor 114 and the image sensor 110, the computer program is loaded into memory 
116. 

The system works by reading off or scanning the raw image data from an 
image pickup device 1 10 for pixels that vary more than a specific amount in intensity 
from their neighboring pixels. The raw image sensor data is the unprocessed brightness 
data output obtained from the image sensor and which has not gone through any lossy 
compression or color processing. The image sensor reads analog voltage or current, 
converts to digital and sends the signal to a host with no fiirther processing or 
compression. The photosites on the image sensor are used to capture either color or 
monochrome digital still or video images. The raw image data is sent to the intelligent 
host over a bus with a data transfer rate which is determined by the bus protocol of the 
particular bus such as a universal serial bus (USB) or a parallel port. 

The raw image sensor data, the location and frequency of occurrence of 
each defective pixel, and the results of all intermediate computations performed by the 
computer program are all stored in memory 116. Other algorithms are then used to 
average the values of an anomalous pixel's neighboring pixels to replace the data from 
the defective pixel. The corrected data can then be fiirther processed and ultimately 
displayed on the monitor 120. The process includes video subsampling, meaning that the 
detection is carried out and repeated at various frame intervals. The video subsampling is 
carried out on one of every 128 (1/128) frames. Altemately, the video subsampling can 
be carried out on every 1/64 video frames. And yet in an additional embodiment, the 
video subsampling is carried out on every l/(n times X) frames, where n is an integer and 
X is not equal to 50 or 60. 50 and 60 correspond to 50 Hz and 60 Hz, which are AC 
lighting frequencies used in the United States and Europe respectively. This way, it is 
ensured that anomalous raw data pixels are not artifacts of the artificial lighting systems. 
The use of video subsampling allows for rapid and optimum corrections without the need 
to scan every frame which would adversely impact the processing speed of the processor 
and the system. Naturally, no video subsampling is employed when detecting and 
correcting defective pixels in still images. 

A statistical database recording the location and the frequency of 
occurrence of defective pixels is generated and stored in memory 1 16 to allow the system 
to learn and adapt its operation over time. The trends from the statistical database are 
stored so that truly anomalous pixels can over time be distinguished from false detection 



of true anomalies in the target image, lighting or other environmentally induced 
anomalies. The operations logic of the statistical database and the details of the operation 
of the computer program are described below. 

FIG. 2 illustrates a simplified flow chart describing the functioning of the 
software program implemented in a system for detecting and correcting defective pixels 
according to one embodiment of the present invention. The first step in defective pixel 
detection and correction includes acquiring a portion of a frame of raw image data from 
an image sensor array, step 210. The raw image data may be data corresponding to a live 
scene being digitized or it may be data corresponding to a calibration or "black" 
background image. The "black" background image may be obtained by reducing the 
integration time of the sensor array or by reducing the gain of the image signal. The raw 
data from the "black" image can be used to detect over active photosites. The raw data 
corresponding to the "black" image must also be black, and if any pixel is not, then it 
corresponds to an overactive photosite. The use of the "black" image can enhance 
defective pixel detection by removing any brightness deviations that originate from the 
scene itself. 

Although the data from an entire frame can be processed at one time for 
defective pixel detection and correction, the algorithm begins its function as soon as data 
from three lines from the image sensor has arrived at the host. Three lines of data from 
the image sensor array will allow for the processing of data for a pixel and all its 
surrounding neighbors. Moreover, when the sensor array is being used to capture video 
images, video subsampling is used so that not every frame of the video captured by the 
image sensor array is required for the defective pixel detection operation. In a preferred 
embodiment, defect detection is carried out at predetermined frame intervals as described 
above and the correction is applied to all video frames. The judicious use of video 
subsampling allows for the method to be very fast and efficient. 

In an alternate embodiment, no video subsampling is used such that the 
defective pixel detection is carried out on every video frame. The use of video 
subsampling is a function of the intelligent host's processor speed. If a processor is fast 
enough to allow detection of anomalous pixels in every frame, then video subsampling is 
not used. If, on the other hand, the processor is not fast enough to allow video processing 
at a desired frame rate, then video subsampling is used to ensure data transfer at that 
desired frame rate. Without video subsampling, anomalous pixel correction is immediate, 
such that defects are corrected in the very frame in which they are detected. With video 





subsampling, anomalous pixel correction is delayed until a frame is sampled to detect 
omalous pixels. Therefore, the choice of using video subsampling, and the rate of 
lib sampling are a function of the processor speed and a trade off between processor 
power and the delay before correction. 
5 After data acquisition, for each of the acquired pixels, the local pixel 

brightness value and also the average brightness value of all its immediately neighboring 
pixels are computed and stored, step 212. Next, a deviation threshold value is 
established, step 214. The deviation threshold establishes the acceptable level of variance 
between a pixel's brightness value and the average brightness value of all its immediately 
10 neighboring pixels. Next, for each acquired pixel, a local brightness deviation is 

computed, step 216. The local brightness deviation is the absolute value of the difference 
'Ij; between a pixel's brightness value and the average of the brightness value of all its 

^ immediately neighboring pixels. 

i;: Next, for each pixel whose data that has been acquired, its local brightness 

15 deviation is compared to the deviation threshold, step 218. Any pixel whose local 
=1=^ brightness deviation exceeds the threshold deviation value is then flagged as a defective 

l^i, pixel. The physical location and the frequency of occurrence of each defective pixel is 

! >^ then recorded in a statistical database, step 220. The statistical database is then queried, 

!/i to determine whether the defective pixel's data value should be corrected, step 222. The 

JS 20 statistical database, by storing the location and frequency of defective pixels, develops 
over time trends which confirm which of the defective pixels warranted correction. The 
logic of the trends from the statistical database initially warrant correction of all flagged 
defective pixels as a default, and over time warrant pixel correction only if a particular 
pixel has an occurrence frequency of at least two out of the last four queries. 
25 The defective pixels that have passed through the statistical database filter 

are corrected next, step 224. In order to correct a defective pixel, the erroneous raw 
brightness data for a defective pixel is replaced by that pixel's local average brightness 
value, which is the average brightness value of all its immediately neighboring pixels. 
Next, the corrected data from the defective pixels as well as data from non-defective 
30 pixels is prepared to be sent for subsequent processing, step 226. Subsequent processing 
may include compression, color processing and encoding to data formats suitable for 
display. The defective pixel detection and correction is carried out on the raw data from 
the image sensor array because it is preferred to correct the data before any subsequent 
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processing has occurred since that processing itself can introduce artifacts which are hard 
to distinguish from artifacts which have been produced as a result of defective photosites. 

The software algorithm is intentionally kept very simple so that the 
processor load and/or the video frame rate is minimally affected. First, the software 
algorithm is kept simple because it only performs three functions, namely the detection, 
correction and statistics fimctions. The statistics routine is kept simple because only 
position information and frequency of occurrence of anomalous pixels are tracked. 
Second, the software algorithm is kept simple so as to have a minimal impact on the rate 
at which data is transferred to the host, so that while a frame is being scanned for 
defective pixels, subsequent frames are not held up. The maximum impact of the 
software algorithm is to at worst reduce the video data transfer rate from 10 frames per 
second (:^s) to 9 fps. Third, the software algorithm is kept simple such that the host 
processor's load is not increased by more 1 % when executing the algorithm. The 
anomalous pixels are detected in one frame and the corrections are later instituted on 
subsequent frames as they are read off the image sensor. 

Therefore, when the system is operating, as raw image data from the image 
sensor is obtained and sent to be processed on a host, frames are captured for defective 
pixel detection and correction as described above. Once a defective pixel has been 
identified and its correction has been warranted by the statistical database, that correction 
will continue for all subsequent frames until the next query is made to the statistical 
database. If at that time the initially flagged defective pixel continues to be designated as 
a defective one, then the corrections will continue as described above. If on the other 
hand, the statistical database does not warrant the correction of defective pixels, then 
those pixels will no longer be corrected until the next query to the statistical database is 
made. The dynamic nature of this system will also allow for corrections of data from 
photosites that become defective over time due to post-manufacture environmental 
changes. 

The statistical analysis segment of the defect detection and correction 
algorithm is an optional one. It is aimed at increasing the efficiency of the pixel 
corrections, so as not to correct anomalies that were not caused by defective photosites, 
and hence save processing time and load. However, the gains in efficiency must be 
balanced against the load imposed by the statistical analysis portion itself. In an alternate 
embodiment not employing the statistical analysis portion, all pixels that get flagged as 
defective get corrected. As in the choice of using or not using the video subsampling, the 
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decision to employ the statistical analysis portion of the defect detection and correction 
algorithm depends on a trade off between efficiency and processor power. If the 
processor is fast enough, then efficiency concerns are not so important. On the other 
hand, if the processor is not fast enough, then pixel correction efficiency becomes 
important enough to warrant the implementation of the statistical analysis portion of the 
defect detection and correction algorithm. 

As will be understood by those of skill in the art, the present invention 
may be embodied in other specific forms without departing from the essential 
characteristics thereof. For example, the rate of video subsampling could be varied, or 
not done at all. Alternately, the logic of the optional statistical database could be altered 
from one of correcting defective pixels as a default to one where defective pixels are 
corrected only if warranted by the trends from the statistical database. Accordingly, the 
foregoing disclosure in intended to be illustrative, but not limiting, of the scope of the 
invention which is set forth in the following claims. 



